Skip to main content
Qlik Resources

Data messages

This topic describes the structure and content of the Data messages produced by the Replicate Amazon MSK endpoint.

Data message
Field Type Description

headers

 Structure Information about the current record.

operation

 Enum The operation type.

Full Load (Replicate transfers the existing records from source table)

REFRESH – insert of a record during Full Load stage.

CDC (Replicate transfers the changes from source table)

INSERT – insertion of new record

UPDATE – update of existing record

DELETE – deletion of a record

changeSequence

 String

A monotonically increasing change sequencer that is common to all change tables of a task.

Use this field to order the records in chronological order.

Applicable to CDC operations.

timestamp

 String

The original change UTC timestamp.

Applicable to CDC operations.

streamPosition

 String

The source CDC stream position.

Applicable to CDC operations.

transactionId

 String

The ID of the transaction that the change record belongs to.

Use this field to gather all changes of a specific transaction.

Applicable to CDC operations.

changeMask

String

Indicates which data columns were changed in the source table.

The change mask is a string of hexadecimal digits, representing a bitmask of data columns in little-endian order. The bit position in the change mask is based on the ordinal of the column in the metadata message of that table.

This means that if there are 10 data columns, they occupy bits 0 to 9 in the bitmask.

If UPDATE mask is 0B hexadecimal, which is 1011 binary – it means that the columns at ordinals 1, 2 and 4 were changed.

The following describes the bit semantics:

  • For INSERT records, all the inserted columns have the associated bits set.
  • For DELETE records, only primary-key (or unique index) columns have the associated bits set. This allows an applier to construct a DELETE statement without having to find the primary key fields from another source.
  • For UPDATE records, each column with a changed value will have the associated bit set.
Information note

LOB columns are not included in the changeMask bit.

columnMask

String

Indicates which data columns are present in the message. Usually, this will include all of the table columns.

Information note

When replicating from an Oracle source without full supplemental logging, some columns might not be present in the data, since they could not be replicated.

The column mask is a string of hexadecimal digits, representing a bitmask of data columns in little-endian order. The bit position in the column mask is based on the ordinal of the column in the metadata message for that table.

This allows the applier to distinguish a null value that is the actual value of the column, from a null value that represents a column which could not be replicated from the source database.

externalSchemaId

String

The Schema ID. This will only be displayed if the Include external Schema IDheader check box is selected.

As the Schema ID changes whenever a DDL is performed on the source table, consumer applications can use this information to determine if the message schema has changed.

transactionEventCounter

Long

The sequence number of the current operation in the transaction.

This can be used to determine the order of operations within a transaction.

transactionLastEvent

Boolean

"True" indicates that it is the final record in the transaction whereas "False" indicates that not all of the records have been processed.

data

 Structure The data of the table record

{columns}

   The column names and values in the current record.

beforeData

 Structure The data of the table record, before the change

{columns}

  

The column names and values, before the change.

Applicable to UPDATE operation.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here